Language and Compiler Support for Out-of-Core Irregular Applications on Distributed-Memory Multiprocessors

Current virtual memory systems provided for scalable computer systems typically offer poor performance for scientific applications when an application’s working data set does not fit in main memory. As a result, programmers who wish to solve “out-of-core” problems efficiently typically write a separate version of the parallel program with explicit I/O operations. This task is onerous and extremely difficult if the application includes indirect data references. A promising approach is to develop a language support and a compiler system on top of an advanced runtime system which can automatically transform an appropriate in-core program to efficiently operate on out-of-core data. This approach is presented in this paper. Our proposals are discussed in the context of HPF and its compilation environment.